REPORT  DOCUMENTATION  PAGE 


Form  Approved  OMB  NO.  0704-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggesstions  for  reducing  this  burden,  to  Washington 
Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington  VA,  22202-4302. 
Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  oenalty  for  failing  to  comply  with  a  collection 
of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


2.  REPORT  TYPE 

MS  Thesis 


1.  REPORT  DATE  (DD-MM-YYYY) 

24-07-2015 


4.  TITLE  AND  SUBTITLE 

Partial  Ordering  and  Stochastic  Resonance  in  Discrete 
Memoryless  Channels 


3.  DATES  COVERED  (From  -  To) 


5a.  CONTRACT  NUMBER 

W911NF-1 1-1-0144 


5b.  GRANT  NUMBER 


6.  AUTHORS 

Roland- Yannick  Kamdem  Kenmogne 


5c.  PROGRAM  ELEMENT  NUMBER 
206022 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAMES  AND  ADDRESSES 

The  University  of  the  District  of  Columbia 
Computer  Science  and  Informati 
Briana  Lowe  Wellman 

Washington,  DC  20008  -1122 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS 
(ES) 

U.S.  Army  Research  Office 
P.O.Box  12211 

Research  Triangle  Park,  NC  27709-2211 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 
ARO 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

58962-NS-REP.23 


12.  DISTRIBUTION  AVAIL1B1LITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  contrued  as  an  official  Department 
of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


14.  ABSTRACT 

In  this  thesis,  we  study  the  performance  of  Discrete  Memoryless  Channels  (DMCs) 
arising  in  the  context  of  cooperative  underwater  wireless  sensor  networks.  We  introduce  a 
partial  ordering  for  the  binary-input  ternary  output  (2,3)  DMC.  In  the  particular  case  of  the 
Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE),  we  use  majorization  theory, 
channel  convexity  and  directional  derivative  in  order  to  obtain  a  partial  solution  to  the  open 


15.  SUBJECT  TERMS 

Partial  Ordering,  Channel  Capcity,  Discrete  Memory  Channel 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

15.  NUMBER 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

ABSTRACT 

OF  PAGES 

UU 

UU 

UU 

UU 

Paul  Cotae 


19b.  TELEPHONE  NUMBER 
202-274-6290 


Standard  Form  298  (Rev  8/98) 
Prescribed  by  ANSI  Std.  Z39.18 


Report  Title 

Partial  Ordering  and  Stochastic  Resonance  in  Discrete  Memoryless  Channels 

ABSTRACT 

In  this  thesis,  we  study  the  performance  of  Discrete  Memoryless  Channels  (DMCs) 
arising  in  the  context  of  cooperative  underwater  wireless  sensor  networks.  We  introduce  a 
partial  ordering  for  the  binary-input  ternary  output  (2,3)  DMC.  In  the  particular  case  of  the 
Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE),  we  use  majorization  theory, 
channel  convexity  and  directional  derivative  in  order  to  obtain  a  partial  solution  to  the  open 
problem  of  partial  ordering  of  the  DMCs.  In  addition,  we  analyze  the  stochastic  resonance 
(SR)  phenomenon  impact  upon  the  performance  limits  of  a  distributed  underwater  wireless 
sensor  networks  operating  with  limited  transmitted  power  and  computational  capabilities.  We 
focus  on  the  threshold  communication  systems  where,  due  to  the  underwater  environment, 
non-coherent  communication  techniques  are  affected  both  by  noise  and  threshold  level.  The 
binary-input  ternary-output  channel  is  used  as  a  theoretical  model  for  the  DMC.  We  derived 
the  capacity  of  the  threshold  (2,3)  DMC  in  the  presence  of  additive  noise.  In  order  to  evaluate 
stochastic  resonance,  we  model  the  theoretical  (2,3)  DMC  as  a  physical  communication 
channel  corrupted  by  additive  noise  with  different  probability  distributions.  The  (2,3)  DMC 
becomes  the  BSC/SE  when  the  probability  density  function  of  the  additive  noise  is  an  even 
function  such  as  Gaussian,  Laplace,  and  Cauchy  distribution.  Due  to  the  complexity  and  the 
non-linearity  of  the  channel  capacity  analytical  expression,  the  Pinsker  and  Helgert  capacity 
bounds  are  also  used  to  evaluate  the  stochastic  resonance  in  the  case  of  the  (2,3)  DMC.  Our 
contribution  consists  in  improving  the  state  of  the  art  on  the  issue  of  partial  ordering  for  the 
(2,3)  DMC  and  deriving  the  optimal  noise  level  required  to  obtain  the  maximum  capacity  for 
a  given  threshold  decision  level  in  the  case  of  the  binary-input  ternary-output  DMC. 


Partial  Ordering  and  Stochastic  Resonance  in  Discrete  Memoryless  Channels 


by 

Roland- Yannick  Kamdem  Kenmogne 


A  thesis  submitted  to  Graduate  Faculty  of  the 
University  of  the  District  of  Columbia 
in  Partial  Fulfillment  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in 
Electrical  Engineering 


Washington,  DC 
May,  2012 


Partial  Ordering  and  Stochastic  Resonance  in  Discrete  Memoryless  Channels 


Roland- Yannick  Kamdem  Kenmogne 


In  this  thesis,  we  study  the  performance  of  Discrete  Memoryless  Channels  (DMCs) 
arising  in  the  context  of  cooperative  underwater  wireless  sensor  networks.  We  introduce  a 
partial  ordering  for  the  binary-input  ternary  output  (2,3)  DMC.  In  the  particular  case  of  the 
Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE),  we  use  majorization  theory, 
channel  convexity  and  directional  derivative  in  order  to  obtain  a  partial  solution  to  the  open 
problem  of  partial  ordering  of  the  DMCs.  In  addition,  we  analyze  the  stochastic  resonance 
(SR)  phenomenon  impact  upon  the  performance  limits  of  a  distributed  underwater  wireless 
sensor  networks  operating  with  limited  transmitted  power  and  computational  capabilities.  We 
focus  on  the  threshold  communication  systems  where,  due  to  the  underwater  environment, 
non-coherent  communication  techniques  are  affected  both  by  noise  and  threshold  level.  The 
binary-input  ternary-output  channel  is  used  as  a  theoretical  model  for  the  DMC.  We  derived 
the  capacity  of  the  threshold  (2,3)  DMC  in  the  presence  of  additive  noise.  In  order  to  evaluate 
stochastic  resonance,  we  model  the  theoretical  (2,3)  DMC  as  a  physical  communication 
channel  corrupted  by  additive  noise  with  different  probability  distributions.  The  (2,3)  DMC 
becomes  the  BSC/SE  when  the  probability  density  function  of  the  additive  noise  is  an  even 
function  such  as  Gaussian,  Laplace,  and  Cauchy  distribution.  Due  to  the  complexity  and  the 
non-linearity  of  the  channel  capacity  analytical  expression,  the  Pinsker  and  Helgert  capacity 
bounds  are  also  used  to  evaluate  the  stochastic  resonance  in  the  case  of  the  (2,3)  DMC.  Our 
contribution  consists  in  improving  the  state  of  the  art  on  the  issue  of  partial  ordering  for  the 
(2,3)  DMC  and  deriving  the  optimal  noise  level  required  to  obtain  the  maximum  capacity  for 
a  given  threshold  decision  level  in  the  case  of  the  binary-input  ternary-output  DMC. 


Acknowledgments 


I  am  thankful  to  my  mentor  and  advisor  Dr.  Paul  Cotae,  P.I.  of  the  Department  of  Defense  (DoD) 
Research  Grant  W91 1NE-1 1-1-0144,  for  his  guidance  and  financial  support  throughout  my 
master  study  and  research. 

My  sincere  thanks  go  to  the  Naval  Research  Laboratory,  in  particular  to  Dr.  Ira  S.  Moskowitz  for 
his  useful  remarks  and  for  his  research  on  top  of  which  my  thesis  is  built. 

I  would  like  to  acknowledge  the  Electrical  and  Computer  Engineering  Department  who  has  been 
helpful  throughout  my  master  degree,  especially  Dr.  Samuel  Lakeou,  Dr.  Sasan  Haghani  and  Dr. 
Esther  Ososanya. 

I  am  especially  thankful  to  my  parents,  Esther  and  Fidel,  my  sisters,  Ginette,  Edith,  Beryl  and  my 
brother,  Roland,  for  their  unending  support,  encouragement  and  spiritual  guidance,  my  friends, 
Laetitia,  Edema,  Garret,  and  Suresh  for  their  encouragement  and  stimulating  discussion  during 
this  research. 

Last  but  not  the  least  I  would  like  to  thank  God,  without  whom  this  would  not  have  been 
possible. 


Contents 

Acknowledgments . iii 

List  of  Figures . v 

List  of  Abbreviations  and  Notations . vi 

Summary . vii 

Chapter  1  Introduction . 1 

1.1  Thesis  Structure . 1 

1.2  Thesis  Significance  and  Contributions . 2 

Chapter  2  Background . 3 

2.1  Communication  Channel . 3 

2.1.1  Entropy,  Conditional  entropy  and  Joint  entropy . 3 

2.1.2  Mutual  information  and  channel  capacity . 4 

2.1.3  Example  of  DMC;  binary  -input  binary-output  (2,  2)  DMC . 7 

2.2  Partial  ordering  and  DMCs . 8 

2.3  Stochastic  Resonance . 10 

Chapter  3  (2,3)  DMC  Mutual  Information  and  Channel  Capacity . 12 

3.1  The  general  (2,3)  DMC . 12 

3.2  Binary  Asymmetric  Channel  with  Symmetric  Erasure  (BAC/SE) . 13 

3.3  Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE) . 14 

3.4  Binary  Erasure  Channel  (BEC) . 16 

3.5  Binary  Symmetric  Channel  (BSC) . 16 

Chapter  4  Partial  Ordering  of  BSC/SE . 19 

4.1  Partial  ordering  and  particular  cases . 20 

4.2  Channel  convexity  and  its  partial  ordering . 21 

Chapter  5  Stochastic  Resonance  in  BSC/SE . 27 

5.1  Physical  communication  channel  and  additive  noise . 27 

5.2  Stochastic  Resonance  and  additive  Gaussian  noise . 32 

5.2.1  Stochastic  Resonance  and  Forbidden  Interval  Theorem  in  the  case  of  the  BSC/SE . 34 

5.2  Stochastic  Resonance  and  capacity  bounds  for  (2,  n )  DMC . 35 

5.3  Optimal  values  for  threshold  and  noise  levels . 38 

Chapter  6  Conclusions  and  future  work . 41 

Appendix . 44 

iv 


List  of  Figures 


Figure  1.  Communication  Channel  Model . 3 

Figure  2.  Relation  between  entropy  and  mutual  information . 4 

Figure  3.  Gaussian  Channel . 5 

Figure  4.  (m,  n)  DMC  representation . 6 

Figure  5.  Physical  communication  channel . 6 

Figure  6:(  2,  2)  DMC  representation . 7 

Figure  7.  (2,  2)  DMC  capacity  as  a  function  of  a  and  /? . 8 

Figure  8.  Geometric  representation  of  71  <  B . 9 

Figure  9.  SR  effect  on  channel  capacity . 10 

Figure  10.  (2,  2)  DMC  Capacity  for  different  values  of  8  =  2,1.5,1-1,1,0.8,0.5,0.3,0.25 . 11 

Figure  11.  Binary-input  ternary  output  (2,3)  DMC . 12 

Figure  12.  Binary  erasure  channel  with  ptl  =  1  —  p21 . 16 

Figure  13.  Binary  Symmetric  Channel . 16 

Figure  14.  BSC/SE  capacity  as  a  function  of  (a,  ft) . 17 

Figure  15.  Representation  of  the  BSC/SE  in  a  plane . 19 

Figure  16.  Partial  ordering  of  Binary  Symmetric  Channel  Capacity . 20 

Figure  17.  Capacity  of  the  Binary  Erasure  Channel . 21 

Figure  18.  BSC/SE  Partial  Ordering  with  a  and  ft  constants . 22 

Figure  19.  BSC/SE  partial  ordering  and  directional  derivative . 25 

Figure  20.  Summary  of  partial  ordering  for  BSC/SE . 26 

Figure  21.  Physical  model  of  binary-input  ternary-output  communication  channel  with  standard 

Gaussian  noise . 27 

Figure  22.  pdf  and  cdf  of  the  Gaussian  noise  with  p  =  0  and  er  =  1 . 28 

Figure  23.  pdf  and  cdf  of  Laplace  noise  with  p  =  0  and  a  =  2 . 29 

Figure  24.  pdf  and  cdf  of  Cauchy  noise  with  location  parameter  x0  =  0  and  scale  parameter  y  =  1 . 30 

Figure  25.  BAC/SE  Capacity  C  of  a  threshold  system  (0  =  2)  with  three  kinds  of  noise . 32 

Figure  26.  Capacity  of  BAC/SE  as  a  function  of  0  and  a . 33 

Figure  27. Capacity  of  BSC/SE  for  0=  {0.5,  0.6,  1,  1.5,  2,  2.5,  3,  3.5,  4} . 33 

Figure  28.  (2,3)  DMC  and  Pinsker  capacity  Bound . 37 

Figure  29.  (2,3)  DMC  and  Helgert  capacity  Bound . 37 

Figure  30.  Capacity(C),  Pinsker  (L)  and  Helgert  (U)  bounds  for  different  values  of  0 . 38 

Figure  31.  Optimal  noise  power  and  threshold  for  stochastic  resonance,  with  a  scalable  factor . 40 


v 


List  of  Abbreviations  and  Notations 


AIT 

AGN 

BAC/SE 

BEC 

BSC 

BSC/SE 

cdf 

(m,  n) 

FIT 

i.i.d 

pdf 

SR 

C(X,  Y) 

%/(Xy) 
H(/) 
H(X) 
I(X,Y ) 

a 

Pij 

M 


Algebraic  Information  Theory 
Additive  Gaussian  Noise 

Binary  Asymmetric  Channel  with  Symmetric  Erasure 
Binary  Erasure  Channel 
Binary  symmetric  Channel 

Binary  symmetric  Channel  with  Symmetric  Erasure 
Cumulative  density  function 

DMC  Discrete  Memoryless  Channel  with  m  inputs  and  n  outputs 

Forbidden  Interval  Theorem 

Independent  and  identically  distributed 

Probability  density  function 

Stochastic  Resonance 

Channel  capacity  with  input  X  and  output  Y 

Directional  derivatives  of  a  function  /(x,  y)  along  the  vector  U 

Hessian  matrix  of  a  function  / 

Entropy  function  of  a  random  variable  X 
Mutual  information  of  two  random  variables  X  and  Y 
n-dimensional  probability  vector 

Probability  of  receiving  the  message  i  giving  that  the  message  j  was  sent 
Stochastic  Matrix 


Summary 


In  this  thesis,  we  study  the  performance  of  Discrete  Memoryless  Channels  (DMCs) 
arising  in  the  context  of  cooperative  underwater  wireless  sensor  networks.  We  introduce  a 
partial  ordering  for  the  binary-input  ternary  output  (2,3)  DMC.  In  the  particular  case  of  the 
Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE),  we  use  majorization  theory, 
channel  convexity  and  directional  derivative  in  order  to  obtain  a  partial  solution  to  the  open 
problem  of  partial  ordering  of  the  DMCs.  In  addition,  we  analyze  the  stochastic  resonance 
(SR)  phenomenon  impact  upon  the  performance  limits  of  a  distributed  underwater  wireless 
sensor  networks  operating  with  limited  transmitted  power  and  computational  capabilities.  We 
focus  on  the  threshold  communication  systems  where,  due  to  the  underwater  environment, 
non-coherent  communication  techniques  are  affected  both  by  noise  and  threshold  level.  The 
binary-input  ternary-output  channel  is  used  as  a  theoretical  model  for  the  DMC.  We  derived 
the  capacity  of  the  threshold  (2,3)  DMC  in  the  presence  of  additive  noise.  In  order  to  evaluate 
stochastic  resonance,  we  model  the  theoretical  (2,3)  DMC  as  a  physical  communication 
channel  corrupted  by  additive  noise  with  different  probability  distributions.  The  (2,3)  DMC 
becomes  the  BSC/SE  when  the  probability  density  function  of  the  additive  noise  is  an  even 
function  such  as  Gaussian,  Laplace,  and  Cauchy  distribution.  Due  to  the  complexity  and  the 
non-linearity  of  the  channel  capacity  analytical  expression,  the  Pinsker  and  Helgert  capacity 
bounds  are  also  used  to  evaluate  the  stochastic  resonance  in  the  case  of  the  (2,3)  DMC.  Our 
contribution  consists  in  improving  the  state  of  the  art  on  the  issue  of  partial  ordering  for  the 
(2,3)  DMC  and  deriving  the  optimal  noise  level  required  to  obtain  the  maximum  capacity  for 
a  given  threshold  decision  level  in  the  case  of  the  binary-input  ternary-output  DMC. 


vii 


Chapter  1  Introduction 


In  this  thesis,  we  discuss  the  partial  ordering  and  Stochastic  Resonance  (SR)  effect  for  the 
binary-input  ternary-output  (2,3)  Discrete  Memoryless  Channels  (DMC).  We  focus  on  the 
Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE)  which  is  a  particular  case  of  the 
general  (2,3)  DMC.  We  establish  the  relations  between  the  general  (2,3)  DMC,  the  Binary 
Asymmetry  Channel  with  Symmetric  Erasure  (BAC/SE),  the  BSC/SE,  the  Binary  Symmetric 
Channel  (BSC)  and  the  Binary  Erasure  channel  (BEC).  For  the  particular  case  of  the  BSC/SE, 
we  provide  a  partial  ordering  under  specific  channel  noise  component  constraints  using 
majorization  theory,  channel  convexity  and  directional  derivatives. 

In  order  to  study  SR,  we  model  the  (2,3)  DMC  by  a  physical  communication  channel.  In 
presence  of  Laplace,  Cauchy  and  Gaussian  noise,  the  SR  phenomenon  is  observed  over  a  certain 
range  of  threshold  values.  We  focus  on  the  additive  Gaussian  noise  in  which  case  we  determine 
the  interval  under  which  we  observe  SR.  In  addition,  the  Pinsker  lower  and  Helgert  upper 
capacity  bounds  are  used  to  study  SR.  The  main  contributions  of  this  thesis  consist  in  deriving 
the  optimal  noise  power  level  required  to  obtain  the  maximum  capacity  for  a  given  threshold 
decision  level  and  introducing  partial  ordering  for  the  particular  case  of  the  BSC/SE. 

1.1  Thesis  Structure 

The  rest  of  the  thesis  is  structured  as  follows: 

Chapter  2  provides  the  background  concepts  and  the  literature  reviews,  which  will  be 
used  throughout  the  thesis. 

Chapter  3  reviews  and  derives  the  mutual  information  and  the  capacity  of  the  (2,3) 
DMC.  It  also  shows  the  relations  between  general  (2,3)  DMC,  the  binary  asymmetric 
channel  with  symmetric  erasure  (BAC/SE),  the  binary  symmetric  channel  with 
symmetric  erasure  (BSC/SE),  the  binary  erasure  channel  (BER)  and  the  binary 
symmetric  channel  (BSC). 

Chapter  4  introduces  the  partial  ordering  for  the  BSC/SE.  The  partial  ordering  is  based 
on  majorization  theory  and  channel  convexity. 
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Chapter  5  analyzes  the  effects  of  SR  and  the  physical  model  of  the  (2,3)  DMC  in 
presence  of  Gaussian,  Laplace  and  Cauchy  noise.  In  addition,  the  optimal  noise  level 
and  the  optimal  threshold  level  in  order  to  obtain  the  maximum  capacity  is  derived. 
Chapter  6  engages  a  discussion  about  the  results  and  future  work. 

1.2  Thesis  Significance  and  Contributions 

This  research  contributes  to  Dr.  Cotae’s  research  titled  “Information  -  Driven  Doppler  Shift 
Estimation  and  Compensation  Methods  for  Underwater  Wireless  Sensor  Networks”,  which  is  to 
analyze  and  develop  noncoherent  communication  methods  at  the  physical  layer  for  target 
tracking  and  use  the  information  theory  tools  to  predict  the  next  target  position  by  processing  the 
data  information  collected  from  a  collaborative  wireless  distributed  sensor  network  [14].  It 
improves  the  understanding  of  the  information  theory  aspects  of  the  physical  communication 
model  in  the  case  of  (2,3)  DMCs.  In  addition,  it  contributes  to  the  improvement  of  the 
performance  limits  of  distributed  underwater  wireless  sensor  networks  operating  with  limited 
transmitted  power  and  computational  environment.  The  results  of  this  work  will  contribute  to  the 
effort  of  overcoming  the  effect  of  noise  in  underwater  sensors  and  signal  processing. 

Precisely,  in  Chapter  3,  we  derive  the  mutual  information  and  the  channel  capacity  of  the 
(2,3)  DMC.  In  Sections  3. 1-3.4,  the  relations  between  (2,3)  DMC,  BAC/SE,  BSC,  and  BEC  are 
established.  In  Chapter  4,  Section  4.2,  we  introduce  a  partial  ordering  for  the  BSC/SE.  Three  new 
theorems  (Theorems  4.2.1,  4.2.2  and  4.2.3),  based  on  channel  convexity  and  majorization  theory, 
are  given  and  are  used  to  provide  a  partial  ordering  for  the  BSC/SE.  In  Chapter  5,  we  evaluate 
the  stochastic  resonance  phenomenon  in  the  BSC/SE  in  presence  of  the  additive  Gaussian, 
Laplace,  and  Cauchy  noise.  In  Section  5.2,  we  use  the  Pinsker  lower  and  Helgert  upper  capacity 
bounds  [13]  to  evaluate  the  stochastic  resonance  phenomenon  for  the  general  case  of  the  (2,3) 
DMC.  In  section  5.3,  the  optimal  values  for  the  threshold  and  noise  levels  resulting  in  maximum 
channel  capacity  are  derived. 
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Chapter  2  Background 


In  this  chapter,  we  provide  a  background  review  on  concepts  used  throughout  the  thesis. 
Mutual  information,  entropy  and  channel  capacity  are  reviewed.  In  addition,  we  review  the 
current  state  of  the  art  in  partial  ordering  and  stochastic  resonance  phenomenon  of  DMC. 


2.1  Communication  Channel 

A  communication  channel,  in  the  Shannon  sense  [1]  is  a  theoretical  model  of  the  physical 
transmission  medium,  as  shown  in  Figure  1,  which  can  be  used  to  model  a  wired  or  a  wireless 
channel.  In  this  model,  the  output  depends  probabilistically  on  the  input.  In  general,  the 
information  associated  with  a  communication  channel  is  characterized  by  a  mutual  information 
function,  a  joint  entropy,  a  conditional  entropy  and  a  channel  capacity,  defined  below. 


Transmitter 


Channel 


Receiver 


Figure  1.  Communication  Channel  Model 


2.1.1  Entropy,  Conditional  entropy  and  Joint  entropy 

The  entropy,  H,  is  a  measure  of  uncertainty  in  a  random  variable  [2].  Mathematically,  the 
entropy  is  defined  as 


H(X)  = -YlXExP(x)log2(p(x))  (1) 

where  A  andp(x)  represent  a  random  variable  and  its  probability  mass  function,  respectively. 
Similar  to  the  entropy,  the  conditional  entropy  and  joint  entropy  represent  the  entropy  of  a 
conditional  distribution  and  a  joint  distribution,  respectively.  For  two  random  variables  X  and  Y, 
the  conditional  entropy  is  defined  as 

H(Y\X )  =  -YlX,yp(x,y)log2(p(y\x))  (2) 

and  the  joint  entropy  is  given  by 

H(X,Y )  =  -I ]x,yp(x,y)log2(p(x,y))  (3) 
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where  p(x,y)  andp(y|x)  represent  the  joint  and  conditional  probability  mass  function  of  the 
random  variable  X  and  Y,  respectively. 

2.1.2  Mutual  information  and  channel  capacity 

The  mutual  information,  /,  is  a  measure  of  the  amount  of  information  one  random  variable 
contains  about  another.  It  also  represents  the  reduction  in  uncertainty  in  a  random  variable  due  to 
another  random  variable  [2].  For  two  random  variables  X  and  Y,  it  is  defined  by 

1(X)Y)  =  H(X)-H(Y\X)  (4) 

From  Figure  2,  we  can  easily  derive  the  relations  between  mutual  information,  conditional 
entropy  and  joint  entropy  as  follow 

H(X,Y)  =  H(Y,X )  =  H(X)  +  H(Y\X)  =  H(X\Y)  +  I(X)Y)  +  H(Y\X)  (5) 

H(X,  Y) 


The  channel  capacity  is  defined  by  Shannon  [1]  as  the  maximum  rate  at  which  one  transmits 
information  over  the  channel  and  recovers  the  information  at  the  output  with  a  vanishingly  low 
probability  of  error.  In  order  to  calculate  the  channel  capacity,  we  maximize  the  mutual 
information  l(X,Y )  over  the  input  distribution  p(x),  x  G  X,  therefore  the  capacity  can  be  written 
as 

C  —  maxp(x)  I  (X;  T)  bits  per  symbol  (6) 
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where  X  and  Y  represent  the  input  and  output  random  variable. 

Depending  on  the  type  of  the  message  transmitted,  a  communication  channel  is  classified  as 
a  continuous  or  discrete  channel.  A  continuous  channel  has  its  transmitted  message  modeled  as 
continuous  signals.  The  most  notorious  continuous  channel  is  the  Gaussian  channel,  illustrated  in 
Figure  3,  where  the  output  at  a  discrete  time  i  is  the  sum  of  the  discrete  input  Xt  and  the 
continuous  noise  Z, .  The  sample  noise  Zj  is  drawn  independent  identically  distributed  (i.i.d) 
from  a  Gaussian  distribution  with  variance  a2  and  zero  mean,  N{ 0,  a2)  [2], 


Xt 


Y, 


Zi~lV(0,a2) 

Figure  3.  Gaussian  Channel 

The  capacity  C  of  the  Gaussian  channel  with  power  signal  constraint  P  and  noise  variance  a2 
is  given  by 


1  P 

C  —  -  log2  (1  +  — r)  bits  per  transmission 


(7) 


Similarly  for  a  Gaussian  bandlimited  channel  we  have 


C  —  wlog2(l  +  bits  per  transmission 


(8) 


where  w  represents  the  bandwidth  of  the  channel. 

Conversely,  a  channel  is  said  to  be  discrete  if  its  transmitted  message  is  modeled  as  a  digital 
signal  as  it  is  depicted  in  Figure  4. 

In  this  thesis,  we  focus  on  the  DMC.  A  discrete  channel  is  said  to  be  memoryless  if  the 
constant  stationary  probability  distribution  of  the  output  depends  only  on  the  input  at  that  time 
and  is  conditionally  independent  of  previous  channel  inputs  or  outputs  [2] .  In  general,  a  DMC  is 
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/Pll  Pnl  \ 

represented  by  a  (m,  ri)  channel  transition  probability  matrix  M  =  I  :  :  I  ,  where 

\Plm  •••  PrnnJ 

each  entry  =  p(y;  |xj)  represents  the  conditional  probability  relation  between  the  output  y, 
and  the  input  Xj.  M  is  a  stochastic  matrix. 


yi 


y2 


yn 


Y 


Figure  4.  (m,  n)  DMC  representation 


In  order  to  study  the  impact  of  noise  on  the  channel  capacity,  we  often  represent  the  DMC  as 
a  physical  communication  model  as  depicted  in  Figure  5,  in  which  the  output  is  the  sum  of  the 
additive  noise  added  to  the  input  during  the  transmission  through  the  channel.  The  physical 
communication  channel  contains  a  threshold  decision  block  t(6)  used  to  convert  the  channel 
output  signal  xt  +  n  into  a  discrete  output  y; . 


Figure  5.  Physical  communication  channel 


In  [22],  an  underwater  acoustic  sensor  network  with  sensors  nodes  grouped  in  clusters  was 
proposed.  The  link  between  each  sensor  and  the  mobile  access  point  is  modeled  by  a  DMC.  The 
binary-input  binary-output  (2,  2)  and  binary-input  ternary-output  (2,3)  DMCs  are  of  great  interest 
for  various  digital  communications  channels,  including  acoustic  underwater  communications  [3]. 
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2.1.3  Example  ofDMC;  binary  -input  binary-output  (2,  2)  DMC 

The  (2,  2)  DMC  is  shown  in  Figure  6.  It  is  characterized  by  the  input  random  variable  X  and 
the  output  random  Y  taking  values  (x1;  x2)  and  (yj,  y2),  respectively.  Its  transition  probability 
matrix  is  defined  by 


M(a,  /?) 


a 

1 

—  a 

a 

a 

fi 

1 

-P. 

P- 

(9) 


where  a,  /?  are  the  conditional  probability  relationships  between  the  input  and  the  output  ( a  — 
p(Y  =  y1\X  =  Xi),/?  =  p(Y  =  y2\X  =  x2)). 


a 


Xi 


*2 


Vi 


yz 


Figure  6:(  2,  2)  DMC  representation 


The  maximum  achievable  rate  or  capacity  of  the  channel  C  over  different  distributions  of  X 
is  given  by 


C  =  maxp(x)  I  (X;  Y)  =  maxp(Xi)  I  (X-,  Y) 


(10) 


where  p(x)  =  p(xj)  =  1  —  p(x2)  is  the  probability  of  sending  the  symbol  xl5  p(x2)  is  the 
probability  of  sending  the  symbol  x2,  and  / (X,  Y)  is  the  mutual  information  between  the  random 
variable  input  X  and  output  Y. 

In  [23],  Silverman  has  shown  that  the  capacity  C:/2  ->  [0,1],  where/2  =[0,1]  x  [0,1],  as  a 
function  of  a  and  /?  is  given  by 


C(a,P) 


log2 


(ah-ip)  -  p  h(ct)  ph(a)-ah(p) 

2  a~P  +  2  a~P 


(11) 


where  C(a,  /?)  =  0  if  a  =  /?  and  h(x)  represents  the  binary  entropy  function  given  by 


7 


and  a  —  1  —  a. 

The  capacity  of  the  (2,2)  DMC  as  a  function  of  a  and  [i  is  illustrated  in  Figure  7. 


Figure  7.  (2,  2)  DMC  capacity  as  a  function  of  a  and  /? 

2.2  Partial  ordering  and  DMCs 

The  partial  ordering  of  DMC  consists  of  comparing  DMC  capacity  based  on  the  noise 
components  without  the  computation  of  the  analytical  relation  of  the  channel  capacity,  which  in 
general,  is  very  complex  and  non-linear.  The  ordering  is  critical,  especially  for  wireless  sensor 
communication  links  at  the  physical  layer  operating  in  very  noisy  environments  with  limited 
computational  and  power  capability  [7].  It  was  first  mentioned  by  Shannon  in  [24]  and  later 
results  were  developed  in  [6],  [7]  and  [25]  for  the  (2,  2)  DMC. 

In  [6],  the  monotonicity  principle  for  convex  functions  is  used  to  geometrically  compare 
binary  DMC  capacity.  In  Figure  8,  the  set  of  binary  DMCs  is  represented  in  the  Cartesian  plane, 
where  each  point  representing  a  channel  is  located  by  its  coordinate  pair  (a,  /?).  Keep  in  mind 
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that  (a,  /?)  represents  the  conditional  probability  relation  between  the  input  and  output  of  the 
channel.  The  set  of  positive  channels  is  then  defined  as  the  set  of  all  channels  such  that  a  —  = 

det(A)  >  0.  In  [6],  it  is  demonstrated  that  the  (2,2)  channel  capacity  decreases  along  any  line 
that  ends  on  a  zero  channel  capacity.  In  order  to  illustrate  the  ordering,  it  is  introduced  the 
relation  <  on  the  set  of  positive  channels.  For  two  positive  binary  channels  A(a,  /?)  and  B(c,  d), 
if  A  <  B  then  C  (A)  >  C  (B),  in  other  terms,  the  capacity  of  channel  A  is  greater  than  the  capacity 
of  channel  B .  Geometrically,  as  depicted  in  Figure  8,  we  have  A  <  B  if  and  only  if  B  is 
contained  in  the  triangle  with  vertices  {(0,0),  (a,/?),  (1,1)}.  Analytically  A  <  B  iff  (A  =  B ) 
or  (a  —  /?  >  c  —  d)  and  (1  —  c)(a  —  /?)>(!  —  a)(c  —  d). 


Figure  8.  Geometric  representation  of  A  <  B 

In  [7],  majorization  theory  is  used  to  partially  order  binary  DMC  under  the  trace  matrix 
constraint.  Cotae  and  Moskowitz  [7]  demonstrated  that  the  binary  DMC  is  Schur  convex  [26]  on 
the  region  Ra  represented  by  the  triangle  ABD  and  Schur  concave  on  the  region  Rb  represented 
by  the  triangle  OAB,  as  shown  in  Figure  8.  Using  these  results,  they  proved  that  for  two  binary 

DMCs,  («!,  1  —  a2 )  =  (Ul  \  ai)and  (/?1;  1  —  /?2)  =  \  f1)  with  the  same  matrix 

V<^2  -I  —  a2/  \P2  I  —  Pi) 

trace  ,  if  (av  1  —  a2)  and  (/?x,  1  —  /?2)  £  Ra  ,  and  (alt  1  —  a2 )  is  majorized  by  (jS1, 1  —  /?2) , 
then  C(a1,l  —  a2)  <  0(^,1  —  p2)  .  On  the  other  hand,  if  (at,a2),  (/?1;/?2)  £  Rt  , 
then  C(a1, 1  —  a2 )  >  C{Plt  1  —  /?2). 
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The  monotonicity  principle  and  the  majorization  theory  proposed  in  [6]  and  [7],  respectively, 
represent  an  approach  and  a  step  forward  in  solving  the  open  problem  of  partial  ordering. 

In  this  thesis,  using  the  majorization  theory,  channel  convexity  and  directional  derivatives, 
we  extend  the  results  obtained  in  [6],  [7]  and  [8]  for  the  case  of  the  (2,3)  DMC. 

2.3  Stochastic  Resonance 

In  general,  the  SR  phenomenon  is  a  non-linear  effect  wherein  a  communication  system  can 
enhance  the  transmission  of  the  information  in  the  presence  of  the  additive  noise  [4],  Various 
performance  metrics  in  the  presence  of  SR  such  as  signal  to  noise  ratio,  mutual  information,  and 
channel  capacity  improvement,  under  a  certain  range  of  power  noise  levels,  were  discussed  in 
[5].  In  Figure  9,  the  channel  capacity  is  plotted  as  a  function  of  the  noise  power  level  [9].  It  is 
observed  that  the  channel  capacity  increases,  as  the  noise  level  increases,  until  the  channel 
capacity  reaches  a  resonance  peak.  This  increase  of  the  channel  capacity  is  not  in  concordance 
with  the  intuition  that  increasing  the  noise  power  level  will  decrease  the  channel  capacity. 


Figure  9.  SR  effect  on  channel  capacity 

In  [9]  and  [10],  the  SR  effect  is  observed  on  the  binary-input  binary-output  (2,2)  DMC 
capacity,  where  maximum  channel  capacity  occurs  at  an  optimal  power  (variance)  of  the  additive 
Gaussian  noise  (AGN)  in  relation  to  a  given  threshold  level  9,  as  depicted  in  Figure  10.  In  [21], 
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the  Pinsker  and  Helgert  bounds,  established  in  [12],  [13],  are  used  to  derive  the  optimal  noise 
power,  for  a  given  threshold,  that  maximize  the  capacity  of  the  (2,2)  DMC. 

In  this  thesis,  following  Chapeau-Blondeau’s  work  in  [5],  we  analyze  the  physical 
communication  model  of  the  (2,3)  DMC  for  threshold  based  SR  due  to  additive  noise.  Due  to  the 
general  non-linearity  of  the  DMC  capacity,  we  will  use  the  Pinsker  and  Helgert  capacity  bounds 
to  study  the  capacity  behavior  of  a  (2,3)  DMC. 


Theta=0.80 
Theta=0.95 
Theta=0.99 
Theta=1 .00 
Theta=1 .01 
Theta=1 .05 
Theta=1 .10 
Theta=1 .20 


0.4  0.5  0.6 

Noise  Power  Level 


Figure  10.  (2,  2)  DMC  Capacity  for  different  values  of  9  =  (2, 1.  5, 1. 1, 1, 0. 8, 0.  5, 0. 3, 0. 25} 


In  this  chapter,  mutual  information,  entropy  and  channel  capacity  are  reviewed.  These 
concepts  will  be  used  throughout  the  remaining  of  the  thesis.  In  addition,  we  assessed  the  current 
state  of  the  art  in  partial  ordering  and  stochastic  resonance  phenomenon  of  DMC. 
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Chapter  3  (2,3)  DMC  Mutual  Information  and  Channel  Capacity 


In  this  chapter,  we  focus  on  the  general  (2,3)  DMC  and  its  particular  cases.  In  Section  3.1,  we 
derive  the  mutual  information  and  capacity  of  the  general  (2,3)  DMC.  In  Sections  3. 2-3. 5,  we 
establish  the  relations  between  the  general  (2,3)  DMC,  the  BAC/SE,  the  BSC/SE,  the  BSC,  and 
the  BEC. 

3.1  The  general  (2,3)  DMC 

The  (2,3)  DMC  {M:  X  ->  Y]  is  characterized  by  a  binary  input  random  variable  X  =  (1,  — 1}, 
a  ternary  output  random  variable  Y  —  (1,0, —1},  and  a  (2  x  3)  channel  transition  probability 

/Pll  P21  P  3l\ 

matrix  M  defined  by  M  =  ( „  „  „  .  The  entries  of  this  matrix  are  the  p,  .■  which 

\P  12  V  22  P32/  7 

represent  the  conditional  probability  between  each  output  and  input  values  as  represented  in 
Figure  11. 


Figure  11.  Binary-input  ternary  output  (2,3)  DMC 


The  probability  matrix  for  the  (2,3)  DMC  channel  is  given  by 

M  = 

(  Pn  =  P(X  =  1\X  =  1)  p2i  =  P(Y  =  0\X  =  1)  p31  =  P(Y  =  -1\X  =  1)  \ 

V  Pi2  —  P(Y  —  1\X  =  — 1)  p22  =  P(Y  =  0|A  =  -1)  p32  =  P(Y  =  -1\X  —  — 1)/ 

Since  M  is  a  stochastic  matrix,  we  have  p31  =  1  —  pxl  —  p21  and  p32  =  1  —  p12  —  p22  ■  By 
defining 
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P(x  =  i)  =  p(*j),  i  =  1,2, 

and 

P(r  =;)  =  p(y,-)  =  1,2,3, 

and  using  (1),  (2),  and  (3)  we  can  rewrite  the  mutual  information  as  follows 

I(X.  r>  =  SuPfey/)  i°92  -0^  (12) 

Since  p(Xj).p(y;- |Xj)  =  p(x(,y;)  and  p(y;)  =  Efc=iP(y/|xfc)  p(xfc)  we  finally  have 

/a.O=  Z?.i  Sj.i  P(*<)  pfa  1^)  tog,  pPlTp(y,l^)  (,3) 

In  order  to  calculate  the  capacity  of  the  (2,3)  DMC  channel,  we  want  to  maximize  the  mutual 
information  I(X,Y).  Based  on  (13),  we  observed  that  I(X,Y)  is  a  function  of  p(xx)  andp(x2) 
since  the  conditional  transition  probability  p(y/  |Xj)  =  py  is  fixed  there,  / (X,  Y)  = 
/(P(xi),p(x2)).  Therefore,  in  order  to  calculate  the  capacity  of  this  channel,  we  need  to 
maximize  I(X,Y )  over  all  distributions  of  the  form  (p {x1 ) ,  p (x2 ) ) .  Since  p(xx)  +p(x2)  =  1, 
maximization  can  only  be  made  over  the  variable  x,  where  x  =  p(xx)  =  1  —  p(x2).  Therefore, 
the  channel  capacity  can  be  written  as 

C  —  max*  I(X,  y).  (14) 

Substituting  (13)  in  (14)  we  obtain 

C  =  max*ri f.1Z^1p(*ap(»l*i)!<>«2  (15) 

While  the  general  solution  for  (15)  is  not  known,  in  [15],  a  closed  form  solution  is  proposed  for 
the  capacity  of  BAC/SE. 

3.2  Binary  Asymmetric  Channel  with  Symmetric  Erasure  (BAC/SE) 

The  BAC/SE  is  a  particular  case  of  the  (2,3)  DMC  with  the  following  constraints:  p21  =  p22 
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and  p  11  >  p12.  Its  transition  probability  matrix  is  given  by 


M  = 


tP  n 
VPl2 


V  22 
V22 


Ps 1\ 

P32J 


(16) 


In  [15],  the  Kuhn-Tucker  condition  is  used  to  find  the  following  solution  for  the  BAC/SE 
channel  capacity  (See  Appendix  for  the  proof): 

p(Vj\x) 


c  =  'L)=iv{yj\x)iog2 


*(p(y_/l*i)-p(y/l*2))+p(y/l*2) 


for  i  —  1,2 


where 


and 


x 


kP32~Pl2 


Pll— Pl2-fc(P31  — P32) 


(17) 


(18) 


_  (PllPllP31P31\ 
\Pl2Pl2P32P32 / 


Pll~Pl2 


(19) 


Using  the  closed  form  solution  (16),  we  will  examine  the  stochastic  resonance  that  arises  in  the 
(2,3)  BSC/SE  channels. 


3.3  Binary  Symmetric  Channel  with  Symmetric  Erasure  (BSC/SE) 

The  BSC/SE  is  a  particular  case  of  the  BAE/SE  with  plx  =  p32.  Since  M  is  a  stochastic 


matrix,  if  plx  =  p32  and  p21  =  p22,  thenp12  =  p31.  Therefore  the  transition  probability  matrix 


of  the  BSC/SE  is  given  by 


M  = 


/P11 

VP12 


P22 

P22 


Pl2\ 

PlJ 


(20) 


We  recall  from  (17)  that  the  capacity  of  the  BAC/SE  is  given  by 


C  =  Y,3j=iP{yj\Xj)log2(- 


p(yj\Xi ) 


x{p{y  px^-piy  j\x2))+p{y  j\x2) 


)  for  i  —  1,2 


For  example,  for  i  =  1  we  have  the  following  expression 
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C  =  P^°32  +  P2ll°92  X(.vJZ)+V22 


(21) 


+P3ilog2 


V  31 

*(P31  -P32)  +P32 


Using  the  following  equalities  pxl  =  p32  and  p22  =  P21  ,  the  capacity  simplifies  to 


C  = 


Puiog2 


p  11 

x(Pll-Pl2)  +  Pl2 


+  Psilog2 


_ P31 _ 

*(P31~P32)  +  P32 


(22) 


We  will  determine  the  value  of  x  that  maximizes  the  mutual  information  of  the  BSC/SE.  From 
(18) and  (19)  we  have 

kP32_Pl2 


X 


Pll-Pl2-fc(P31-P32) 


with  k 


_  ( PllVllP31P31 
^Pl2^12P32^322 


Pll“Pl2 


Using  the  previous  constraints,  when  applied  to  the  BSC/SE  p12  =  p31  and  p21  =  p22  and  pxl  = 


p32  ,  we  have 


Therefore  (22)  becomes 


k  =  1 


and 


P32~P32  _  1 

2(Pll— P32)  2 


c  =  Pulog2 


2Pn 


P11+P3: 


+  P3l(0p2 


P31 


P11+P31 


(23) 

(24) 


It  is  interesting  to  note  that  the  capacity  of  the  BSC/SE  is  function  only  of  two  variables  just  as  in 
the  (2,  2)  DMC.  If  we  set  pxl  =  a,  p31  —  b,  c  =  p12  and  d  =  p32  then  we  can  re-write  both  M 
and  C  (X,  Y)  for  the  particular  case  of  the  BSC/SE  as  follows 


and 


/ a  1  —  a  —  b  b\ 
Vb  1  —  a  —  b  a/ 


(25) 


With  1  >  a,  b  >  0  and  a  +  b  <  1. 

Next,  we  will  focus  on  the  binary  erasure  channel. 
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3.4  Binary  Erasure  Channel  (BEC) 

The  BEC,  shown  in  Figure  12,  is  a  particular  case  of  the  BSC/SE  with  the  particularity  that 
P12  =  P31  =  0-  Replacing  p12  =  p31  =  0  in  (25)  the  capacity  of  the  BEC  becomes 

C  ~  Pnl°92  ~  —  P11  —  1  —  P21  (26) 

Pll 

which  corresponds  to  the  well-known  capacity  of  the  binary  erasure  channel  [2].  Fori  =  2  in 
(19),  following  the  same  steps,  we  obtain  the  same  result  for  C. 


P11 


Figure  12.  Binary  erasure  channel  with  pxl  =  1  —  p2 1 
3.5  Binary  Symmetric  Channel  (BSC) 

The  BSC,  depicted  in  Figure  13,  is  a  particular  case  of  the  (2,  2)  DMC  but  can  also  be 
derived  from  the  BSC/SE  when  p21  =  p22  =  0. 
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Proof:  Given  that  M  is  a  stochastic  matrix,  if  P21  —  P22  —  0  then  p31  =  1  —  Pn,  therefore 
replacing  p31  =  1  —  plx  in  (24),  we  have 

C  =  p11log22p11  +  (1  -  p11)log2 2(1  -  Pn)  (27) 

=  L°g2(.2pn)Pl1  +  log2{ 2(1  -  Pn))1_Pl1 

=  (o^2(2p11Pll(l  -  Pn)1_Pll)>  so 


C  =  1  -  (—Pnlog2 (pn)  -  (1  -Pn)/op2(l  -pn))  =  1  -  H(Pn)  (28) 


The  above  relation  corresponds  to  the  channel  capacity  of  the  BSC. 

In  Figure  14,  we  plot  the  BSC/SE  (25)  in  function  of  the  noise  elements  a  and  b.  In  the  same 
Figure,  we  illustrate  the  BSC,  which  occurs  when  a  —  1  —  b  and  the  BEC,  which  occurs 
when  b  —  0. 


Figure  14.  BSC/SE  capacity  as  a  function  of  (a,  b ) 
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In  summary,  we  have  derived  the  mutual  information  and  the  channel  capacity  of  the  general 
(2,3)  DMC.  In  addition,  the  solution  of  the  capacity  of  the  BAC/SE  is  used  to  establish  the 
relationships  between  the  BAC/SE,  the  BSC/SE,  the  BSC,  and  the  BEC.  We  proved  that  the  BSC 
and  the  BER  are  all  particular  cases  of  the  general  (2,3)  DMC.  The  results  of  this  chapter  are 
summarized  in  Table.  1. 


Table  1:  Summary  of  (2,3)  DMCs 


Channel 

M 

Constraints 

Channel  Capacity 

(2,3) 

general 

DMC 

( a  1  —  a  —  b  b\ 
Vc  1  —  c  —  d  dJ 

BAC/SE 

f a  1  —  a  —  b  b\ 
Vc  1  —  c  —  d  d / 

a  >  b 

1  —  a  —  b 

—  1  —  c 

-  d 

V-fvl x)loc  (  V{y*Xi)  1 

j-[  1  1  2  \x(p(yj |afr)  -p(yj |ac2))  +p(y;- \x2)J 

BSC/SE 

/ a  1  —  a  —  b  b\ 
Vc  1  —  c  —  d  d ) 

a  >  b 
a  —  d 
b  —  c 

2  a  2b 

alog2  +  blog2 

a+ b  a+ b 

BSC 

/a  1  —  a  —  b  b\ 
Vc  1  —  c  —  d  d / 

_  /  a  1  —  a\ 

~  VI  —  a  a  / 

a  +  b  =  1 
a  —  d 
b  —  c 

1  —  h(a ) 

BEC 

fa  1  —  a  0\ 

VO  1  -  a  0/ 

b  —  c 
=  0 
a  —  d 

a 
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Chapter  4  Partial  Ordering  of  BSC/SE 


In  this  section,  we  focus  on  partial  ordering  in  the  case  of  the  BSC/SE.  The  BSC/SE  is  of 
interest  because  in  [16],  it  is  proven  that  in  presence  of  additive  Gaussian  noise  (AGN),  the 
general  (2,3)  channel  becomes  the  BSC/SE  (2,3).  Our  partial  ordering  is  based  on  channel 
convexity  and  majorization  theory  [261.  In  Section  4.1,  we  review  the  partial  ordering  for  the 
particular  case  of  the  BSC  and  BEC.  In  Section  4.2,  we  prove  three  theorems  (Theorems  4.2.1, 
4.2.2  and  4.2.3)  based  on  channel  convexity  and  majorization  theory.  These  theorems  are  used  to 
provide  a  partial  ordering  for  the  BSC/SE. 

We  recall  from  (25)  that  C(a,  b )  =  alog2  +  blog2  with  1  >  a,  b  >  0  and, 

a  +  b  <  1;  we  will,  given  the  noises  resources  a  and  b,  determine  the  channel  with  the  higher 
capacity  without  computing  (25)  or  using  the  Moskowitz  approximations  [12]. 

In  Figure  15,  we  represent  the  set  of  BSC/SEs  in  the  Cartesian  plane.  The  capacity 
expression  of  the  BSC/SE  is  defined  over  the  triangle  OAB .  The  triangle  OAB  corresponds  to  the 
constraints  that  apply  to  the  noise  elements  a  and  b,  that  is,  0  <  a,  b  <  1  and  a  +  b  <  1.  Using 
(25),  we  can  easily  verify  C(a,b)  —  C(b,a ).  Note  that  along  the  AC  segment  (y  =  x),  the 
BSC/SE  capacity  is  null  (zero  channel  capacity).  Similar  along  the  OA  segment  we  have  the 
BEC  and  along  the  AB  segment  we  have  the  BSC  (y  =  1  —  x).  In  Figure  15,  the  points  A (1,0) 
and  B  (0,1))  represent  the  noiseless  BSC/SE  channel. 


B(0,1)  C(l,l) 
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4.1  Partial  ordering  and  particular  cases 

In  [7],  a  partial  ordering  for  the  BSC  was  proposed.  For  a  BSC  with  a  crossover  probability  a 

and  a  transition  matrix  M  —  a  ^  ^  a^,  w^en  a  >  2  anc^  a  increasinS’  the  capacity  is 

1 

increasing  with  a  minimum  capacity  of  0  bits  per  transmission  (zero  channel  capacity  a  —  -)  and 

1 

a  maximum  of  1  bit  per  transmission  (noiseless  channel  a  —  1).  By  symmetry,  when  a  <  -  and 

a  is  decreasing,  the  capacity  of  the  BSC  is  also  increasing  with  a  minimum  capacity  of  0  bits  per 

1 

transmission  (zero  channel  capacity  a  —  -)  and  a  maximum  of  1  bit  per  transmission(noiseless 

channel  a  —  1).  Therefore  in  order  to  partially  order  BSCs,  all  one  needs  is  to  compare  the 

1 

crossover  probability.  For  two  BSCs  with  crossover  probability  %  and  a2 ,  if  a1>  a2>  -  , 
then  C(alt  1  —  ax)  >  C(a2, 1  —  a2 )  ,  as  depicted  in  Figure  16,  similarly,  if  %  <  a2  <  -  then 
C(alt  1  —  ax)  >  C(a2, 1  —  a2). 


Figure  16.  Partial  ordering  of  Binary  Symmetric  Channel  Capacity 


For  the  BEC  with  a  probability  of  erasure  1  —  a  and  a  transition  matrix  M  —  ^  ^ 

the  capacity  of  the  BEC  is  an  increasing  function  of  a  as  depicted  in  Figure  17.  Therefore  for 
two  BECs  with  probabilities  of  erasure  1  —  at  and  1  —  a2,  if  %  >  a2,  then  C(alf  1  —  ax)  > 

C  (a2>  1  —  u2). 


20 


Figure  17.  Capacity  of  the  Binary  Erasure  Channel 


4.2  Channel  convexity  and  its  partial  ordering 

Our  approach  of  partial  ordering  is  based  on  partial  derivatives,  channel  convexity  and 
majorization  theory.  We  will  focus  in  the  01 A  triangle,  shown  in  Figure  15.  The  partial  ordering 
is  done  under  each  of  the  following  constraints  on  the  noise  elements:  a  is  constant,  b  is 
constant,  a  +  b  is  constant,  and  a  —  b  is  constant.  The  same  result  could  be  applied  to  the 
channel  capacity  in  the  OIB  triangle  because  of  the  symmetry  of  the  BSC/SE.  We  introduce  the 
following  theorems. 

Theorem4.2.1:  For  BSC/SEs  defined  by  the  transition  probability 

matrix  ^  ^  ^  i  =  1,2,  if  at  is  constant,  and  bt>  b2,  then  C^a^bf)  < 

C(ai,  b2).  Conversely  if  bt  is  constant  and  %  >  a2,  then  C(at,  b{)  >  C(a2,b{). 

Proof:  In  order  to  prove  the  above  theorem  we  will  find  the  partial  derivatives  of  the  channel 
capacity.  We  have 
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Given  that  we  are  in  the  01 A  triangle,  we  have  a  >  b  or  2a  >  a  +  b  which  imply  that 

dC(a,b) 

da 


so  for  any  fixed  b ,  the  function  C (a,  b)  is  strictly  increasing;  therefore  for  h  —  constant , 
if  %  >  a2  then  C (a1;  b)  >  C  (a2,  b ).  Similarly,  we  have 


dC(a,b ) 
db 


=  l0»2  o  <  0. 


therefore  for  a  fixed  a  the  function  C (a,  b)  is  strictly  decreasing,  therefore  for  a  =  constant,  if 
bx  >  b2  then  £(<2,^)  <  C(a,b2 ). 


The  above  theorem  is  summarized  in  Figure  18.  For  any  BSC/SE  (a^bi)  included  in  the 

polygon  formed  by  the  intersection  of  the  horizontal  and  vertical  segment  from  (a1;  bt)  to  both 

1  1 
the  y  =  x  line  (for  a  <  -)  or  the  y  =  x  line  and  the  y  =  1  —  x  line  (for  a  >  -),  C^a^bi)  < 

C^a^bi). 


B(  0,1)  C(l,l) 


Figure  18.  BSC/SE  Partial  Ordering  with  a  and  b  constants 
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From  these  results,  the  partial  ordering  is  done  as  follows;  given  two  BSC/SE  transition 


stochastic  matrices  Mt  = 


1-ai-bi 
1  -a-i-bi 


and  Mx 


1  —  a1  —  b± 
1  —  ax  —  bx 


if  cij  <  % 


and  bj  >  then  C  {at>  bf)  <  C  (a1;  b x) . 


The  next  partial  ordering  theorem  is  based  on  channel  convexity,  directional  derivative  and 
majorization  theory. 


Definition  4.2.1:r271 

/ 


A  function  f(x1,x2, ...  ,xn)  is  convex  if  its  (n  x  n)  Hessian  matrix 

d2f  \ 


H  (/)  = 


dx\  dXj^dXn 

d2f  d2f 

\^dxndx1  dXn 


) 


is  positive  semidefinite;  that,  is  the  eigenvalues  of  the  H(/) 


matrix  are  non-negative. 

Theorem  4.2.2:  The  BSC/SE  is  a  convex  function. 


Proof:  Based  on  Def  4.2.1  we  will  obtain  the  Hessian  matrix  of  the  BSC/SE  and  compute  its 
eigenvalues.  We  have 


d2C(a,b )  _  b 
da 2  a2+ab 


and 


d2C{a,b )  _  a 
d b2  b2+ab 


d2C(a,b)  _  -1 
dadb  a+b 


and 


d2C{a,b )  _  -1 
dbda  a+b 


Therefore 


H(C)  = 


and  the  eigenvalues  of  H(C)  are 


log  10(2) 


A  —  0  and  A  = 


b 

_  1  \ 

a2+ab 

a+b 

1 

a 

a+b 

b2+ab / 

a2+b2 

ab(a+b)  log(2) 


>  0 


(29) 


(30) 
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Therefore  the  BSC/SE  is  a  convex  function.  This  motivates  using  majorization  theory  for  partial 
ordering  of  (2,3)  BSC/SE. 


Definition  4.2.2:  [26]  A  real-value  function  0  defined  on  the  set  of  n-dimensional  probability 
vectors  is  said  to  be  Shur  convex  (Shur  Concave)  if  0  is  order  preserving  (inverse  order 
preserving)  with  respect  to  the  partial  order  c.  That  is  if 


a  <  b  =>  0(a)  <  4>(b)  (a  <  b  =»  0(a)  >  0(b))  (31) 


Definition  4.2.3:  [27]  The  directional  derivative  D^f(x,  y) ,  of  a  multivariate  differentiable 
function  fx(x,y )  along  a  given  unit  vector  ~u  at  a  given  point  (x,  y)  represents  the  instantaneous 
rate  of  change  of  the  function  moving  through  (x,  y)  in  the  direction  of  H 


£>u/(x.  y)  =  ei 


5/(x,y)  ,  „  df  (x,y) 
dx  9y 


where  u(elt  e2)  is  a  unit  vector  [27]. 


(32) 


We  will  re-write  (25)  as 


C  (x,  y)  =  xlog2  ^  +  y log2  ^ 


(33) 


in  the  (0,  x,  y,  z)  Euclidian  3-space  where  1  >  x  >  y  >  0  and  x  +  y  <  1.  We  define  the  region 
Rg,  constituted  of  the  set  (x,  y),  such  that,  1  >  x  >  y  >  0  and  x  +  y  <  1 


Theorem4.2.3:  For  BSC/SEs  defined  by  the  transition  probability 

(CL;  1  —  CL;  —  bi  b;\ 

matrix  Mt  —  I  ,  .  ,  ,  I,  i  —  1,2,.,  if  at  +  bi  is  constant  then  the  capacity  function  of 

\D(  1  CL[  Dj  u  j/ 

the  BSC/SE  is  Schur  convex. 


Proof:  Our  proof  is  based  on  directional  derivative.  One  can  easily  prove  using  vector  calculus 

that  for  any  two  points  (x1;  k  —  x-J  and  (x2,  k  —  x2)  G  RG,  such  that  xx  >  x2,  the  unit  vector  of 

11 

the  line  x  +  y  =  k  is  u(—  ,  ^=).  Similarly,  if  (x-l  <  x2),  then  the  unit  vector  along  the  line 
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1  1  _ 

x  +  y  =  k  is  ~  j=)-  Therefore  the  directional  derivative  of  C(x,  y)  along  it  at  any  given 

point  (x,  y)  is  given  by 


DUC (x, y)  =  ^  log2  Q  <  0  (34) 

From  (34),  we  can  conclude  that  the  capacity  decreases  along  any  line  of  equation  x  +  y  =  k  in 
the  direction  of  w.  In  other  terms,  for  any  two  points  (xx,  yx)  and  (x2,y2)  such  that  xx  +  yx  = 
x2  +y2  and  xt  >  x2,  that  is,  if  (x^yj)  >  (x2#y2),  then  CCx^y^  >  C(x2fy2).  This  proves  that, 
if  x  +  y  is  constant,  then  the  capacity  is  Shur-convex. 

1  1 

By  defining  v(-^=,-^=),  as  the  unit  vector  of  the  line  x  —  y  =  k  ,  the  previous  approach  could 
be  used  to  prove  the  following  theorem: 

Theorem  4.2.4:  For  any  (x1;  yx),  (x2,  y2)  G  RG  such  that  xx  —  yx  =  x2  —  y2  if  xx  <  x2, 
thenCCx^)  >  C(x2fy2). 

Theorems  4.2.3  and  4.2.4  are  summarized  in  Figure  19,  where  the  capacity  decreases  in  either 
direction  of  u  or  F. 


Figure  19.  BSC/SE  partial  ordering  and  directional  derivative 


25 


The  results  from  Theorem  4.2.1,  4.2.3  and  4.2.4  are  represented  in  Figure  20.  These  results 
improve  the  partial  ordering  previously  discussed.  For  any  (dj,  hj)  includes  in  the  polygon,  as 

shown  in  Figure  20,  with  vertices {(%,  hx),  (1+a*~&1 ,  1~a*+*>1)  ,(  ,i  ,  (felt  fej  j ,  //(a^bi)  > 
C (cij,  bj).  Analytically  this  translates  into 


{Ltj  \ 

b.  >  bi  then  CCa^bi)  >  C(ai7  bj)  (35) 

at  —  bi  <  —  bx 


Figure  20.  Summary  of  partial  ordering  for  BSC/SE 


Example:  Consider  two  BSC/SEs  with  respective  noise  matrices 


0.1 

0.1 


/0.5  0.2 
V0. 3  0.2 


.  Which  channel  has  the  highest  capacity? 


We  have  0.6>0.5  and  0.2<0.3;  therefore  without  computing  (27),  we  have  C(A)  >  C (B). 


In  conclusion,  this  chapter  provides  a  partial  ordering  for  BSC/SE  under  several  noise 
component  constraints  a  and  b,  which  represents  the  conditional  probability  relationships 
between  the  inputs  and  the  outputs  of  the  channel.  We  partially  order  BSC/SE  when  a  or  b  is 
constant.  In  addition,  we  prove  that  when  a  +  b  is  constant,  the  capacity  function  of  the  BSC/SE 
is  Schur-convex  for  which  case  we  use  majorization  theory  to  partially  order  BSC/SE. 
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Chapter  5  Stochastic  Resonance  in  BSC/SE 


In  this  chapter,  following  the  work  of  [17],  we  examine  the  effect  of  additive  Gaussian  noise, 
Laplace  noise,  and  Cauchy  noise  on  the  capacity  of  the  (2,3)  BSC/SE  derived  in  (25).  We 
assume  that  noise  has  a  Gaussian,  Laplace,  and  Cauchy  distribution,  all  with  zero  mean  (zero 
location  parameter  for  Cauchy  distribution).  The  physical  communication  channel  is  used  to 
model  the  (2,3)  DMC  and  to  derive  the  probability  transition  matrix  components  p(J  of  the  (2,3) 
DMC  in  presence  of  the  Gaussian,  Laplace,  and  Cauchy  noise.  The  SR  is  observed  for  the  (2,3) 
DMC  in  presence  of  Gaussian,  Laplace,  and  Cauchy  noise.  In  Section  5.1,  we  focus  on  the 
particular  case  of  the  AGN,  for  which  case  we  use  theorems  derived  in  Section  4.2  to  determine 
the  forbidden  interval  [20]  of  the  BSC/SE  capacity.  In  Section  5.2,  the  Pinsker  lower  and  Helgert 
upper  capacity  bounds  are  used  to  evaluate  the  SR  in  the  BSC/SE.  In  Section  5.3,  the  optimum 
threshold  value  that  maximizes  the  channel  capacity  of  the  BSC/  SE  is  derived. 

5.1  Physical  communication  channel  and  additive  noise 

The  physical  model  is  represented  in  Figure  21,  where  in  the  presence  of  the  additive  noise  n, 
and  the  input  signal  s,  the  received  signal  y  is  given  by 

y  —  s  +  n  (36) 

The  input  s  takes  the  binary  values  of  -1  and  1,  with  the  probabilities  p(st)  —  p(s  = 
—  1)  and p(s2)  =  p(s  —  1). 


Input  s  =  {_[ 

n 

Figure  21.  Physical  model  of  binary-input  ternary-output  communication  channel 

with  standard  Gaussian  noise 


+ 

X 


TresholcL  t(0) 


Output  y  = 
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The  physical  communication  channel  contains  a  threshold  decision  block  function  t(9) , 
where  9  represents  the  threshold  value.  Based  on  the  threshold  level  9  and  the  noise  level,  the 
received  signal  is  converted  into  a  discrete  random  variable  y  taking  on  the  values  of  1,  0,  or  -1. 
For  the  ternary  output,  the  threshold  function  is  defined  as  follows: 

1—1  s  +  n  <  —9 

t(0)  =  0  -9  <s  +  n<9  (37) 

(  1  s  +  n>  9 

By  using  the  threshold  level,  the  analytic  expression  for  each  of  the  conditional 
probabilities  pxl  p2\,  P31,  V12,  Pt2>  P32  between  the  binary  input  and  the  ternary  output  in 
presence  of  additive  Gaussian,  Laplace,  and  Cauchy  noise  will  be  determined. 

We  start  by  giving  a  brief  review  of  the  pdf  and  the  cdf  of  the  Gaussian,  Laplace  and  Cauchy 
noise. 

•  Gaussian  Noise:  The  Gaussian  noise  with  mean  p  =  0  and  variance  a  >  0  has  the 
following  probability  density  function  and  cumulative  density  distribution,  respectively.  In 
Figure  22,  we  plot  the  pdf  and  the  cdf  for  Gaussian  noise  with  p  =  0  and  a  —  1. 


Figure  22.  pdf  and  cdf  of  the  Gaussian  noise  with  p  =  0  and  <7=1 


28 


Laplace  noise:  Laplace  noise  with  fj.  —  0  and  variance  a  —  2/? 2  has  the  following 
probability  density  function  and  cumulative  density  distribution,  respectively.  In  Figure  24, 
we  plot  the  pdf  and  the  cdf  for  Laplace  noise  with  fi  —  0  and  a  —  2. 


-|n| 


/(n)  = 

II 

(40) 

1 

\\eP 

e  <  o 

<e)  = 

e  >  o 

(41) 

Figure  23.  pdf  and  cdf  of  Laplace  noise  with  ju  =  0  and  a  =  2 


Cauchy  noise:  The  Cauchy  noise  with  location  parameter  x0  =  0  and  scale  parameter  y  has 
the  following  probability  density  function  and  cumulative  density  distribution  respectively. 
In  Figure  24,  we  plot  the  pdf  and  the  cdf  for  Gaussian  noise  with  x0  =  0  and  y  —  1 


f(n ) 


1  Y 
n  n2+Y2 


(42) 


Fn(6>)  =  ^  arctan  Q  +  \ 


(43) 


Figure  24.  pdf  and  cdf  of  Cauchy  noise  with  location  parameter  x0  =  0  and  scale  parameter  y  =  1 


The  following  results  are  valid  for  any  of  the  distributions  described  above. 

Following  the  physical  model  depicted  in  Figure  21,  we  have 

pxl  =  P(1\1)=P(Y  —  1|S  =  1)  =  Pr(s  +  n>0|s  =  l)  =  l  —  Pr(n  <  —  1  +  0)  = 

l-Fn(-l  +  0)  (44) 

Using  a  similar  approach  we  have 

(P21  =  P(0|1)  =  Fn(-1  +  0)  -  Fn(- 1  -  0) 
p31  =  P(-l\l)  =  Fn(-1  -  0) 

"  P12  =  P(U  —  1)  =  1  —  Fn(l  +  0)  (45) 

p22  =  P( 0|  —  1)  =  Fn(l  +  0)  -  Fn(  1  -  0) 

{P32  =  p(-i\-i)  =  Fna-0) 


Theorem  5.1.1:  The  (2,3)  DMC  becomes  BSC/SE  when  the  pdf  of  the  additive  noise  is  an  even 
function. 

Proof:  Given  that  the  pdf  is  an  even  function,  one  can  easily  verify  that  F(— x)  —  1  —  F(x). 

Then  according  with  (44)  and  (45)  the  following  relation  stands 

Pll  —  P32>  P21  —  P22>  P31  —  P12  (46) 
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Since  Fn  (0)  is  a  strictly  continuous  and  increasing  function  1  —  0  >  —  1  —  0  ,  we  obtain  that 
Fw(  1  —  0)  >  Fw(— 1  —  0)  which  corresponds  to 


Pn  >  P12  (47) 

Equations  (46)  and  (47)  are  the  relation  constraints  of  the  BSC/SE. 

At  this  point,  we  have  extended  the  results  of  [16]  by  showing  that  the  (2,3)  physical 
communication  channel  presented  in  Figure  21,  is  a  BSC/SE  channel  in  the  presence  of  any  noise 
that  has  an  even  probability  distribution  function.  This  further  motivates  our  work  to  study  the 
capacity  SR  behavior  of  this  channel  which  was  not  presented  in  [19]. 


Using  (44)  and  (45),  we  can  re-write  the  (2,3)  DMC  matrix  as: 


/Pll  P  21  P3l\ 
VPl2  P22  P32 / 


(1  -  Fw(— 1  +  0)  Fw(— 1  +  0)  —  Fw(— 1  —  0)  Fw(— 1  -  0)\ 
VI  —  Fn(1  +  0)  Fn(1  +  0)  —  Fw(l  —  0)  Fw(l  —  0)  / 


By  using  (25)  the  capacity  of  the  BSC/SE  channel  could  then  be  rewritten  below  in  (49)  and  it  is 
plotted  in  Figure  25  for  Gaussian,  Laplace  and  Cauchy  noise  and  a  trehshold  0  =  2. 

c(X'Y)  =  F*(1  -  Wog2  + 

FN(-t  -  e)log2  (0,(r„(_1.,yF;1-!)9)>rM(1.8))  (49) 

So  far,  we  have  shown  that  the  (2,3)  DMC  in  presence  of  some  type  of  noise  exhibit 
stochastic  resonance.  The  range  value  for  the  threshold  and  the  noise  power  in  order  for  the  (2,3) 
DMC  to  exhibit  stochastic  resonance  and  maximize  the  channel  capacity  is  of  interest. 
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Figure  25.  BAC/SE  Capacity  C  of  a  threshold  system  (6  =  2)  with  three  kinds  of  noise. 

In  the  next  section,  in  order  find  the  range  over  which  the  (2,3)  DMC  exhibit  SR,  we  will 
focus  on  the  additive  Gaussian  noise,  Pinsker  and  Helgert  capacity  bounds. 

5.2  Stochastic  Resonance  and  additive  Gaussian  noise 

We  start  by  re-plotting  (49)  in  Figure  26  as  a  function  of  the  threshold  9  and  the  variance 
er(noise  power).  Since  C(X,  Y)  is  a  function  of  9  and  er,  we  will  re-write  it  as  follow, 

C(X,Y)  =  Cg(a)  (50) 

We  observe  the  phenomenon  of  SR  on  capacity  over  a  specific  range  of 
9  and  noise  power  a.  In  Figure  27,  we  plot  the  capacity  of  the  BSC/SE  for  different  values  of 
the  threshold  9.  To  be  more  specific,  in  Figures  26  and  27,  the  SR  appears  only  for  9  >  1.  For 
the  noise  power  o  G  [5,10]  and  the  thhresold  level  9  >  3,  the  capacity  is  vanishing  to  zero. 
Similar  results  were  reported  in  [4],  [5],  and  [9]  for  the  (2,2)  DMC.  While  these  results  are  in 
concordance  with  the  intuition  that  increasing  the  noise  power  on  the  channel  will  decrease  the 
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Capacity(X,Y)) 


capacity,  the  opposite,  when  the  noise  power  decreases  the  capacity  is  also  decreasing  as  it  will 
be  detailed  in  the  next  section. 


sigma  "  teta 


Figure  26.  Capacity  of  BAC/SE  as  a  function  of  0  and  a 


theta= 

■  theta= 

■  theta= 

■  theta= 

■  theta= 
theta= 

■  theta= 
theta= 

■  theta= 

■  theta= 


4 

=3.5 
=3.0 
2.0 
1  .5 
=  1  .2 
=  1  O 
=0.8 
=0.5 
=0.1 


5 

Sigma 


Figure  27. Capacity  of  BSC/SE  for  0=  {0.5,  0.6, 1, 1.5,  2,  2.5,  3,  3.5,  4} 
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5.2.1  Stochastic  Resonance  and  Forbidden  Interval  Theorem  in  the  case  of  the  BSC/SE 

In  [20],  the  Forbidden  Interval  Theorems  (FIT)  in  stochastic  resonance  are  introduced.  The 
FIT  gives  the  necessary  and  the  sufficient  condition  for  stochastic  resonance  to  occur.  In  general, 
the  forbidden  interval  is  the  region  or  interval  in  which  there  is  no  stochastic  resonance.  A 
similar  approach  is  reported  in  [21],  where  the  Algebraic  Information  Theory  (AIT)  was  used  for 
the  (2,  2)  DMC  to  determine  the  forbidden  interval.  Given  the  close  relation  between  partial 
ordering  and  the  AIT,  we  will  follow  the  steps  of  [21]  to  confirm  our  previous  observation  for 
the  forbidden  interval  in  the  case  of  the  BSC/SE. 

Definition  5.2.1  [21]  A  threshold  system  exhibits  stochastic  resonance  (as  a  function  of  er)  if  and 
only  if  there  exists  0  <  a1  and  0  <  er2  such  that  oy  <  er2  and  Cg(af)  <  Cg(of) 

Theorem  5.2.1:  The  BSC/SE  exhibits  stochastic  resonance  if  and  only  if  the  treshold  6  <£  [0,1]. 

Proof:  We  follow  closely  the  proof  used  in  [21]  for  the  case  of  (2,  2)  DMC.  We  will  write  the  cdf 

f2 

1  Q  _ £ _ 

of  the  Gaussian  distribution  with  zero  mean  (Fn(0)  =  e  2a2  dt)  as  the  cdf  of  the 

standard  normal  cdf  (<t>(0)  =  f_m  e~~ ).  By  a  change  of  variables  we  obtain  Fn(9)  —  <t> 

•  For  9  £  [0  1)  as  o  increases,  since  1  —  9  <  0  we  have  a  =  (~~)  decreases  and 

)  increases.  When  a  decreases  and  b  increases,  Theorem  4.2.1  tells  us  the 

capacity  decreases.  Therefore  for  9  £  [0  1) ,  as  er  increases,  the  capacity  decreases; 
therefore  it  does  not  exhibit  stochastic  resonance. 

•  For  9  —  1,  as  o  increases  a  —  <P  =  0.5  and  b  =  1  —  0  (^j  increases.  When  a  is 

constant  and  b  increases.  Theorem  4.2.1  tells  us  the  capacity  decreases.  Therefore  for 
9  —  1,  as  o  increases,  the  capacity  decreases;  therefore  it  does  not  exhibit  stochastic 
resonance. 
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•  For  6  >  1  ,  as  er  — >  0+  ,  a  — >  0  ,  b  — >0  therefore  C(a,b )  — >  0  .  When  er  — >  oo  , 

1  1 

a  — >  -  b  — >  -  therefore  C(a,b )  — >  0  .  Since  there  exist  er  >  0  such  that  a  =£  b  meaning 
that  C(a,  b )  ^  0,  we  can  conclude  that  the  capacity  increases  for  some  inteval  of  er. 


5.2  Stochastic  Resonance  and  capacity  bounds  for  (2 ,n)  DMC 

Due  to  the  non-existence  of  a  closed  form  expression  for  a  (2,  n)  channel  capacity  in  general 
and,  the  non-linear  expression  of  the  (2,3)  channel  capacity  in  particular,  we  will  use  the  bounds 
(upper  and  lower)  developed  in  [12]  for  the  particular  case  of  the  (2,3)  DMC  in  order  to  examine 
the  phenomenon  of  stochastic  resonance. 

In  [12], [13]  it  has  been  proven  that  any  (2,  n)  DMC  with  2  inputs  and  n  outputs  with  the 

transition  matrix  M  — 

pn2  —  1  —  Pi  2  —  "•  Pn- 1,2  ■>  can  be  approximated  by:  a  lower  bound  called  the  Pinsker  bound, 
that  we  denote  by  L,  and  upper  bound,  the  Helgert  bound,  that  we  denote  U.  The  Pinsker  and 
Helgert  bounds  are  defined  as  follows: 

L  <  C  <  U 

where 

n-l  \  -i2 

^  \Pi,l  “Pul  )  +  IPn,l  Pn,2  I 
i  =  1  / 


L  = 


8  ln(2) 


VPl,2 


Pn— 1,1  and 


U  =  max 


:  Pi,  2) 


For  a  (2,3)  DMC,  the  above  expressions  are  reduced  to 

1  o 

[IPn  -  P12 1  +  IP21  -  P22 1  +  IPsi  -  P32 1]2  <  max(pn  +  p21;  p12  +  p22)  - 
min(pli;  p12)  -  min(p21;  p22)  (51) 


We  have  shown  previously  that  using  the  threshold  model  given  in  Figure  21  and  modeling  noise 
as  a  standard  Gaussian  distribution  we  have  the  following  equality: 
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Pll  —  P32>  P21  —  P22  >  P31  —  Pl2<  P21  —  V22<  Pll  —  Pl2 >  *ind  P32  >  P31 
Using  these  relations  and  (51)  we  have 

2  ln(2-)  (P11  —  P12  )2  —  C  —  Pii  ~  P12  (52) 

Using  (44)  and  (45)  we  have 

^  (Fw(  1  -  0)  -  Fw(-1  -  0)  )2  <  C  <  Fn(1  -  0)  -  Fw(-1  -  0)  (53) 

Finally,  we  have 

L  =  -  9)  -  F»(-!  -  9)  )2  =  /ce, CT)  (54) 

U=  Ft,(l-fl)-Fff(-l-fl)  =  9(8,ff)  (55) 

The  Pinsker  lower  capacity  bound  L  and  the  Helgert  upper  capacity  bound  U  are  plotted  in 
Figure  28  and  29,  respectively.  In  Figure  30,  the  lower  and  the  upper  capacity  bounds  along  with 
the  capacity  of  the  BSC/SE  are  plotted  for  different  values  of  the  threshold  0.  We  noted  that,  just 
as  mentioned  in  [9],  the  bounds  capture  the  increasing/decreasing  behavior  of  the  capacity  and 
exhibit  stochastic  resonance  for  the  threshold  level  0  >  1. 

Following  similar  work  done  in  [21],  we  will  use  the  bounds  to  find  the  optimum  noise  given 
a  threshold  that  will  maximize  the  channel  capacity.  The  above  lower  and  upper  bounds  are 
given  by: 

L=  ^j(^(l-0)-^(-l-0))2andU=  Fw(l  —  0)  —  Fw(— 1  —  0). 
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Figure  28.  (2,3)  DMC  and  Pinsker  capacity  Bound 


Figure  29.  (2,3)  DMC  and  Helgert  capacity  Bound 


37 


Sigma 


Sigma 


theta=1 


Sigma 

theta  =2 


Figure  30.  Capacity(C),  Pinsker  (L)  and  Helgert  (U)  bounds  for  different  values  of  0 


5.3  Optimal  values  for  threshold  and  noise  levels 

The  optimal  values  of  correlated  noise  and  threshold  level  are  of  great  interest.  As  in  [21], 
our  goal  is  to  find  the  critical  points  of  the  capacity  bounds.  In  order  to  find  the  critical  points,  we 

derive  the  partial  derivatives  ^L(0,  cr)  and  ^U(0,cr)  of  the  lower  and  upper  bounds,  and  find 

the  solution  to  the  equations  L(0,  cr)  =  0  and  ^U(0,  cr)  =  0  ,  respectively. 

The  partial  derivative  of  the  bounds  with  respect  to  cr  will  have  the  same  critical  points 
because: 

=  j^2)  -  0)  -  fn{- i  -  0))'(fn(  i  -  0)  -  fn(- i  -  0)) 

Thus,  solving 

d 

—  L(0,  cr)  =  0 
da 

is  equivalent  to 


d 

—  U(0,  cr)  =  0 
da 
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which  implies  that  they  both  have  the  same  critical  points.  Therefore,  we  only  focus 
on^U(0,<7)  =  Fn(  1  -  0)  -  Fn(-  1  -  0). 

We  have 

(0,a)  =  ^(Fw(l  -  0)  -  Fn(—1  -  0)  )  = 

d  1  r1-0  t2  _1  — (i— o')2  -(-i-g)2 

=  -r - j=  e  2o2  dt  —  ——=  [(1  —  0)e  2o-2  —  ( —  1  —  0)e  202  ] 

do  <jy[2n  J_i_g  a2\2n 

Solving  for  er,  U (0,  a)  —  0,  we  obtain 


For  0  >  1  and  a  >  0,  we  finally  have: 


o  = 


29 


(57) 


The  above  relation  gives  us  the  optimum  power  noise  o  as  a  function  of  9.  In  Figure  31,  we  plot 
(57),  a  =  /(0),  in  blue  and  its  inverse,  /-1(0),  in  red.  The  plots  depicted  in  Figure  31  provide 
theoretically  two  opportunities  to  improve  the  channel  capacity,  either  by  adding  or  reducing 
noise  depending  on  the  threshold  or  by  simply  adjusting  the  threshold  depending  on  the  noise 
level.  It  is  important  to  mention  that  from  the  plot  of  Figure  31,  we  see  that  as  0  increases,  a  -> 
0  .That  is  because  lime^  a  —  0  =  0 ,  which  means  that  0  =  a  is  an  oblique  asymptote  to 
a  =  /(0)  and  0  =  /-1(cr) 
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Figure  31.  Optimal  noise  power  and  threshold  for  stochastic  resonance,  with  a  scalable  factor 


In  this  chapter,  we  have  evaluated  the  SR  phenomenon  in  the  BSC/SE  in  presence  of  additive 
Gaussian,  Laplace  and  Cauchy  noise.  In  the  particular  case  of  the  Gaussian  noise,  the  forbidden 
interval  is  determined.  Due  to  the  non-linearity  and  the  complexity  of  the  analytical  expression 
of  the  channel  capacity,  we  use  Pinsker  lower  and  Helgert  upper  capacity  bounds  to  also  evaluate 
SR  in  BSC/SE,  for  which  the  optimal  noise  power  in  function  of  the  threshold  value  is  obtained. 
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Chapter  6  Conclusions  and  future  work 


In  this  thesis,  we  have  introduced  a  partial  ordering  for  the  BSC/SE  which  is  a  particular  case 
of  the  BAC/SE  and  (2,3)  DMC.  We  analyzed  the  general  (2,3)  DMC  and  established  the 
relations  between  the  BAC/SE,  BSC/SE,  BSC  and  the  BEC.  We  demonstrated  that  in  presence  of 
the  noise  with  even  probability  distribution  function  the  general  (2,3)  DMC  becomes  the 
BSC/SE.  Our  partial  ordering  is  based  on  channel  convexity,  majorization  theory  and  directional 
derivative.  We  are  able  to  give  a  partial  ordering  for  the  BSC/SE  under  specific  conditional 
probability  between  channel  input  and  channel  outputs  constraints. 

In  addition,  we  studied  the  threshold  based  stochastic  resonance  behavior  of  binary-input 
ternary-output  DMC.  The  physical  communication  channel  model  is  used  to  analyze  the  channel 
capacity  behavior  of  the  BSC/SE  in  the  context  of  the  stochastic  resonance  phenomenon.  In 
presence  of  Gaussian  noise,  Laplace  noise  and  Cauchy  noise,  the  BSC/SE  exhibit  SR.  For  the 
particular  case  of  the  Gaussian  noise,  the  results  of  the  partial  ordering  are  used  to  identify  the 
forbidden  interval.  Due  to  the  complexity  and  the  non-linearity  of  the  general  (2,  n)  DMC 
capacity,  the  Pinsker  lower  and  Helgert  upper  capacity  bounds  are  used  to  analyze  SR  for  the 
(2,3)  DMC.  Similar  to  the  BSC/SE  capacity,  the  capacity  bounds  exhibit  SR.  Based  on  these 
bounds,  we  analytically  derived  the  optimum  noise  power  level  depending  on  the  threshold  level 
that  will  maximize  the  (2,3)  DMC  capacity. 

The  results  of  this  thesis  clearly  indicate  how  one  could  compare  channel  capacity  under  the 
condition  that  applied  without  using  the  analytical  relation  of  the  capacity  or  the  capacity 
bounds.  They  also  demonstrate  how  one  could  improve  the  capacity  of  the  binary  threshold 
communication  channel  by  changing  the  noise  power  or  the  threshold  level. 

In  this  thesis,  the  focus  was  on  the  Gaussian  noise,  Laplace  noise  and  Cauchy  noise,  all  with 
zero  mean  fi  =  0  (x0  =  0  for  Cauchy  noise).  In  future  research  we  will  extend  these  results  to 
noise  distribution  with  different  mean  values  (different  location  parameters).  In  addition  we  will 
use  more  practical  underwater  noise  models  to  study  the  SR  phenomenon  in  the  (2,3)  DMC. 
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Appendix 


Derivation  of  the  Channel  Capacity  of  the  Binary  Asymmetry  Channel  with  Symmetry  Erasure 

In  this  section,  we  use  the  Kuhn-Tucker  conditions  to  derive  the  analytical  solution  for  the 
capacity  achieving  input  probability  distribution  for  Binary  Asymmetric  Channel  with 
Symmetric  Erasure  (BAC/SE). 

We  recall  from  Chapter  2  that  the  channel  capacity  of  the  (2,3)  DMC  is  characterized  by  a 
binary  input  random  variable  X  —  {xl,x2},  a  ternary  output  random  variable  Y  —  {yi,  y2, 3/3}  and 

{IP  11  P2I  P3l\ 

a  (2  x  3)  channel  transition  probability  matrix  M  defined  by  M  —  ^  J.  ^ie  cnlr'cs 

of  this  matrix  are  the  p^-  —  p(y;|Xj)  which  represent  the  probability  or  receiving  the  symbol  yj 
given  that  the  symbol  xt  was  sent.  We  have  shown  in  (15)  that  the  capacity  of  the  (2,3)  DMC  is 
defined  by 

C  =  max,  S=1  l%lP(xd  v{y,\Xi)log2  (A.l) 

where  p(xx)  =  1  —  p(x2)  =  x 


The  BAC/SE  is  a  particular  case  of  the  (2,3)  DMC  with  p21  —  V22 

Theorem  A.l:  [15]  An  input  distribution  given  by  {p(x1),p(x2)}  achieves  capacity  if  and 
only  if  there  exists  a  constant  C  such  that 

z?-ii nil°ai  =  c  ■ vi  with  p(xd  >  0  (A'2) 

1  PU1°B  2  £  c  ■ vi  wilh  pw  = 0  <A-3> 

For  i  =  1,  we  can  rewrite  (A.2)  as  follow 


C  =  p11log2 


V 11 


p(Xi)pn  +  p(x2)pi2 


+  P2ilog2 


V  21 


P(*l)P21  +P(.X2)P22 
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+P31^°$I2  p(Xl)p31+p(x2)p32 


(A.4) 


For  i  —  2,  (A. 2)  becomes 


C  =  p12log2 


V 12 


p(Xi)pn  +  p(x2)pi2 
+P3ilog2 


+  Pz2^og2 


P  22 


POi)P2i  +P(x2)p22 


P  32 


P(^l)P31  +  Pfe)P32 


(A.5) 


Since  p21  —  P22>  by  equaling  (A.4)  to  (A.5),  we  obtain 


_ PnPllP3iP31 _ 

[x(pn  -  p12)  +  Pi2]pn[x(p31  -  p32)  +  p32]p 31 


_ Pl2Pl2P32P32 _ 

[X(P11-P12)+P12]P1Z[X(P31-P32)+P32]P32 


(A. 6) 


For  simplicity,  we  define 


m  =  x(pxl  -p12)  +p12,n 


*(P3i  -  P32)  +  Ps 2  and  t 


Pl2PlZP32P32 

PnPllp3lP31 


Replacing  m,  n  and  t  in  (A.6),  we  obtain 


mPl2-PllnP32-P31  -  t 


(A.7) 


Given  that  M  is  a  stochastic  matrix,  we  have, 


Pn  +  P21  +  Psi  =  P12  +  P22  +  P32  =  1  (A.  8) 

Since  p21  =  p22,from  (A.8),  we  have 

P12  “Pn  =  P32 -Psi,  (A.9) 

Combining  (A.9)  and  (A.7),  we  finally  obtain 
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1 


(A.  10) 


n  ^ - 

—  =  fpn— P12 
m 


Replacing  m,  n  and  t  by  their  relations  in  (A.  10),  we  obtain, 


*(P31~P32)+P32  _  f Pl2Pl2P32V32\Pll~Pl2 

x(Pll~Pl2)+Pl2  VpnPllp3lP31/ 


(A.  11) 


Solving  (A.l  1)  for  x,  we  obtain 


P11P11P31P311P11_P12 


X  = 


P12— P32l  i2p12p32p32j 


(A.  12) 


(Pl2-Pll)-(P32-P3l)(p“p12p32p32)Pl1  Pl2 


The  relation  (A.  12)  represents  the  input  probability  distribution  that  achieves  the  capacity.  With 
x  found,  one  obtain  the  analytical  solution  of  the  BAC/SE  by  replacing  (A.  12)  in  to  (A.l). 
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